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2. 


1.  Introduction 

A  practical  problem  in  which  the  results  of  the  present 
paper  may  be  useful  is  as  follows.  The  experimenter  is  inter¬ 
ested  in  a  certain  event,  say  A  ,  defined  in  terms  of  behavior 
of  the  subject  (e.g.,  choice  of  one  of  the  two  displayed  stimuli 
which  are  to  be  discriminated) . 

The  experimenter  needs  the  probability  P  (A)  ,  where  s 

s 

denotes  the  subject.  To  estimate  this  probability  one  should 
make  a  series  of  independent  observations  on  the  same  subject, 
a  procedure  often  not  feasible  for  reasons  such  as  memory  or 
learning  effects,  or  simply  subject's  boredom. 

In  such  cases,  one  often  makes  the  assumption  that  Pg (A) 
does  not  depend  on  s  ,  and  proceeds  to  estimate  P(A)  using 
data  for  large  groups  of  subjects,  each  tested  once  or  twice. 

The  crucial  issue  in  such  a  procedure  is  the  inter-subject 
variability  of  Pg(A)  •  The  present  paper  gives,  among  others, 
methods  of  assessing  this  variability,  and  testing  the  hypothesis 
of  equality  of  average  Pg (A)  in  two  groups  of  subjects,  under 
the  constraint  that  at  most  two  observations  are  taken  for  each 
subject.  In  a  sense  this  paper  constitutes  a  statistical  counter¬ 
part  of  an  earlier  paper  written  by  one  of  the  authors  (see  Barto- 
szynski  ( 1978) ) . 


2.  The  General  Scheme 

We  shall  consider  the  following  situation.  Let  Gn  =  {G^,...,Gn} 
be  a  system  of  independent  experiments.  Assume  that  each  experiment 


3. 


may  lead  to  a  "success"  or  to  a  "failure",  with  (unknown) 
probability  of  success  in  experiment  equal  p^  .  Our 

problem  will  be  to  construct  methods  of  inference  about  pro¬ 
babilities  p^  in  situations,  when  for  some  reasons  one  is 
allowed  to  make  at  most  two  independent  observations  of  each 
experiment  . 

Obviously,  not  much  may  be  inferred  about  the  individual 
values  p^  ;  we  shall  therefore  construct  estimators  of  "moment¬ 
like"  quantities 


i  n 

p  =  —  1  p. 

^  n  .  L ,  *i 
i=l 


n  2  _2 

»  iii  -  5 


i  n 

=  1  l 

r*  ^ 


(l) 


Let  be  1  or  0  depending  on  whether  the  first 

trial  in  experiment  Gi  leads  to  success  or  failure,  and 
similarly,  let  Y  be  the  random  variable  coding  the  outcome 
of  the  second  trial  on  . 

Vie  assume  that  different  experiments  and  different  trials  on 
the  same  experiment  are  independent;  moreover. 


P(X.  =  1)  =  P(Yi  =  1)  =  pi  ,  i  =  1 . n  .  (2) 


Denote 


Vj 


(3) 


and  put 


U 


1 

n 


n 


l 

i=l 


Yi 


(4) 


4. 


(5) 


L  =  i$(U  +  V) 


n 


i  Jx 


w  =  z  -  uv 


The  following  proposition  is  an  improvement  of  the  similar 
proposition  proved  by  Bartoszyfiski  (1971) : 


PROPOSITION  1.  The  random  variables  L  and  W  are  unbiased 

2 

estimates  of  p  and  a  respectively.  Moreover ,  Var  L  £  l/8n  , 
Var  W  <  3/4n  and  Var(w|o^  =  0)  £  l/16n  . 

Proof;  By  (2)  ,  we  have  EXi  =  EYi  =  pi  .  Consequently, 

1  n 

EU  =  EV  =  —  7  p.  =  p  ,  and  also  EL  =  p  .  Since 

n  i=l  1 

2 

E£  =  EX . Y .  =  EX.EY.  =  p.  ,  we  get,  by  independence  of  U  and  V 
Mi  ii  ii  i 


n 


EW 


=  1  1  E^.. 

n  i=l  11 


1  n  2 
EUV  =  A  l  p 
n  i=l  1 


t  n  7 
EUEV  =  -  l  p  -  p 
n  i£1  pi 


(6) 


=  a 


which  proves  the  unbiasedness. 

Next,  Var  *i  =  Var  Y±  =  Pi(l  -  pi>  <  1/4  ,  and  using  the 
assumption  of  independence,  we  can  write 
i  n 

Var  U  =  Var  V  =  -  7  Var  X.  <  l/4n.  Since  U  and  V  are 

2.i  1 

n^  i=l 

independent,  we  have 


Var  L  =  \ (Var  U  +  Var  V)  s  l/8n  . 


(7) 


5. 


It  remains  to  evaluate  the  variance  of  W  .  We  may  write 


Var  W  =  Var  (  -  T  £ . .  -  UV  ) 
n  1=1  11 


,  n  ?  n 

~  T  Var  £ . .  +  Var  UV  -  -  Cov  (7  £ .  .  ,  UV  ) 
n2  i=l  11  n  1=1  11 


n  n  n 

=  2  I  Var  £  +  ^  Var  £  £  -  Cov  (  £  £  , 

n  i=l  11  n  i ,  j  =  l  1J  nJ  i=l  11  k, 

=  -T  I  Var  £  +  l  Cov  (£  £  ) 

n  i=l  1  n  i,j,k,m=l  J 


n  i,k,m=l 


^  Cov  <  5ii'  ‘•kn,  >  • 


Taking  into  account  the  fact  that  Cov  (£..,£,  )  is  zero 

ij  km 

unless  i  =  k  and/or  j  =  m  ,  we  obtain  after  some  transformations 
the  formula 


Var  W  = 


U?.--.  A 

_  4 


“  J,  C°V  ^ii'W 


_  4 


l  l  Cov 

1=1  1*1  11  1J 


+  -7  l  l  Cov  ^ir  ^ii* 

n  j=l  1/i  1J 


+  4  I  l  I  c°v  (£  ,,  £  k)  . 

n4  iai  J*i  k/i  iK 


We  have,  if  different  letters  stand  for  different  indices. 


Ct — :  6 


6. 


Cov 

^ii) 

=  Pj(l  - 

PI>  ' 

(10) 

Cov 

(tll' 

=  PjPid 

"  Pi}  ' 

(11) 

Cov 

(?ij. 

Cij) 

=  PiPjd 

-  PiPj)  • 

(12) 

Cov 

(5lJ. 

^ik) 

=  PjPkPi 

(1  -  P±)  • 

(13) 

Thus,  all  terms  in  (9)  are  nonnegative,  and  all  are  bounded 
from  above  by  1/4  .  We  may  therefore  write,  omitting  the  second 
sum  in  (9) 


Var  W  < 


In  -_1)  .  n 

n4 


n(n  -  1)  _2_  n(n  -  1)  (n  -  2) 

4  +  4  *  4 

n 


< 


(14) 


which  gives  the  asserted  bound. 

2 

Finally,  if  a  =  0  ,  then  all  pi  are  equal.  Let  their 
common  value  be  p  .  Substituting  (10)  -  (13)  to  (9),  we  obtain, 
after  reduction 


Var  (Wjo2  »  0)  =  n  -2X  p2  (1  -  p)2  .  (15) 

n 

The  maximum  is  attained  at  p  =  \  and  is  asymptotically  equal 
to  l/16n  ,  which  completes  the  proof. 

Thus,  the  bound  for  the  conditional  variance  Var (W jo  =  0) 
is  sharp.  One  may  conjecture  that  the  general  bound  for  Var  W 
can  be  improved,  even  considerably .  However,  simple  numerical  data 
show  that  the  maximum  under  constraint  o  -  0  is  not  the  overall. 


maximum.  Indeed,  if  n  =  6  and  p.  =  . . .  =  p  =  0.5  (hence 

1  D 

2  -3 

o  =  0)  ,  the  variance  of  W  equals  l/16n  =  8.68  •  10  .  If 

2 

p,  =0.45  ,  p„  =  ...  =  p_  =0.5  ,  p,  =0.55  (hence  a  = 

1  £  J  o 

8.3  •  10-4)  variance  of  W  is  8.69  •  10~3  .  If  p^  =  p2  = 

p^  =  0.45  ,  p4  =  p<-  =  p^  =  0.55  (hence  =  2.5  •  10-3) 

variance  of  W  is  8.71  •  10-3  .  Similarly,  if  p^  =  p2  = 

o 

P3  =  0.4  ,  p4  =  p^  =  p6  =  0.6  (hence  o'  =  0.01),  we  have 
Var  W  =  8.8  •  10-3,  etc. 

To  make  Proposition  1  useful  for  practical  applications, 
we  shall  complement  it  by  specifying  conditions  under  which 
the  distribution  of  W  is  asymptotically  normal 

For  that  purpose,  we  assume  that  the  system  is  a 

beginning  of  a  potentially  infinite  sequence  of  independent  ex¬ 
periments  G^,  G2<...  .  In  short,  we  assume  that  n  may  be 
chosen  arbitrarily  large. 

We  shall  prove 

PROPOSITIOl!  2 .  If 

°°  ^ 

1  P?(l  -  P~)  =  °°  (16) 

i=l  1  1 

then  the  random  variable  W  defined  by  (5)  is  asymptotically 
normal. 

2  2 

Proof.  Since  p^d  -  P^)  *  2p^{l  -  pi)  ,  we  have  also 

oo 

l  p.  (1  -  p.  )  =  «  (17) 

i=l  1  1 


and  it  follows  that  all  three  random  variables  U,  V  and  Z 
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defined  by  (4)  and  (5)  are  asymptotically  normal  (see  e.g. 

Fisz  1961) - 

We  shall  prove  that  the  joint  distribution  of  (Z,U,V) 
is  asymptotically  normal.  Consider  a  linear  combination 

aZ  +  bU  +  cV  (18) 


which  may  be  written  as 


i  n  -I  ** 

-  V  [aX.Y.  +  bX.  +  cY.)  =  -  T  n. 
n  . L.  li  l  l  n  .  S  i 

x=l  1=1 


n 


(19) 


Each  of  the  random  variables  9^  assumes  the  values  0,b,c, 
and  a  +  b  +  c  with  probabilities  respectively  equal  to 
(1  -  P±)2  ,  Pi(l  -  /  PA(1  -  Pi)  and  p2  .  Consequently 


Er^  =  (a  +  b  +  c)pi  +  (b  +  c)Pi(l  -  pA 


;) 


(20) 


and  also,  after  some  reductions, 

Var  rii  =  a2p2  (1  -  p2)  +  2a  (b  +  c)  p2  (1  -  p^)  +  (b2  +  c2)  p.^  (1  -  p^  .  (21) 

Since  the  random  variables  are  bounded,  it  suffices  to  show 

that  l  Var  diverges  for  any  a,b,c  which  do  not  vanish 

simultaneously.  If  a  /  0  ,  this  is  true  in  view  of  (16) ,  while 
if  a  =  0  ,  we  must  have  b  +  c  >  0  ,  and  divergence  is  ensured 
by  (17)  . 

Using  now  theorem  from  Rao  (1965),  p.  319-340,  we  conclude 
that  the  random  variable  W  is  asymptotically  normal,  with  mean 
o2  and  variance  bounded  by  3/4n  . 
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Let  us  now  consider  two  disjoint  (hence  independent)  systems 
G  ^  and  G  ^  ,  consisting  respectively  of  n^  and  TI2  exper¬ 
iments.  Let  n  =  n^  +  and  let  G  =  G^  u  G^  be  the  com¬ 

bined  system. 

In  the  sequel,  we  shall  denote  by  and  the 

summations  extended  over  the  indices  in  G^  and  G^  respect¬ 
ively. 

Let 


-(1)  _  I  v*1) 
P  -  ~  l 


=  i  P.  ,  p(2)  =  1  I(2)P. 

n . L  i  '  *  n.~L  1 

1  2 


-p<2’>2 


(22) 

(23) 


-  2 

and  let  p  and  a  have  the  same  meaning  as  before. 

Under  the  constraints  of  the  preceding  section,  namely  that 
one  can  have  at  most  two  independent  trials  on  each  experiment, 
we  shall  construct  a  test  for  the  hypothesis  that  p^  =  p^  . 

Let  W  ,  and  W2  be  the  random  variables  defined  by  (5) 

for  systems  G,  G ^  and  G^2^  . 

We  shall  prove 


10. 


-U)  =  p(2)  .  MnrP.over,  _if  n  ->  -  ,  then_for^n)L  V  n2 

^nln2 

we  have  Var  K  <  cn  3 

- -  n 

Proof ■  Let  us  write 


2 


■;I^i  '  p> 

=  I  y*13  (P.  -  p(1)  +  +  P)2  +  n  ^  (pi 

n  1 


p(2»  *  p<2) 


pi2 


n  1 


-  (2) s  2 


ni 


+  -r-  (P 


p(1)  -  p>2  *  ^  (p(2)  - 5,2 


(25) 


n2  ,  =  <2> 


.  !!i  02  +  »2  *  ^  (pu)  -  p>2  *  V  (p 

-  n  1  n  2  n 


-v  2 

P> 


By  Proposition  1  the  random  variables  W  *  W1  and  W2 

,  2  2  and  02  respectively.  Conse- 

unbiased  estimators  of  o  .  2 


quently  < 


ni  ^2.  EW  =  -i(p{1)  -  p>*  +  -7f(p 

EK  =  EW  -  EWj_  -  n  EW2  n 


«ith  equality  holding  if  and  only 
_  (1)  -  _  -  (2) 

To  evaluate  the  variance  of  K  .  let  us  write 

w  -  i  ieu  -  «v  “  i  ^ii  -  £  l5Jk  ■ 


=,2  .  ^rs‘2>  -  p)2  a  0  -  <26> 

if  p^1)=  P^  '  since  then 


(27) 


and  simi 


ilar  formulas  hold  for  Wj_  and  W2  - 


Using  (27)  we  may  write 


K  =  I  {X  y  dd  +  ^  I  (2)  ^jk  '  n  S  ^jk1 

K  n  1n]_  J*  n2  jK  J 


Consequently,  we  obtain 


n  Var  K  = 


i  .  v(1> 


<iu’  «,k-  r- 11  V 


t  x  Cov  (J<2>  5Jk.  lm  SJk)  +  V°V  I'l*’ 


Cov  (£  1  ^jk'  "  nn2 


Cov  (I(2)^jk'  I&jk*  ' 


since  and  ^  ^jk 


are  independent. 

ily  those  covariances  Cov  Ujk'  Z, 


which  are 


By  taking  into  account  only  those  covariance  "JK' 

are  not  aero.  we  arrive,  after  considerable  algebra,  at  the 


formula 


n  Var  K 


C2_/  TU>  o2U  -  P?)  *  ^  I'"'  P?U 


=  (— )  V  Pi11  - 

*  nn  V  1 


‘1  1 


2  i 


t  4(212-,  jd)  T  U)  PjP2U  -  Pi> 

nnl  i 

n.  2  (2)  ,  12)  , 

*  'i  l  pjPi  1  ‘  Pi 

n  2  i  j^1 


'l  y(l)  yd>  p.p2(l  -  pt)  - 


12. 


^  l{2)  J(1,pjpJu  -  pt> 

n  n2  i  J 


n2i  r  (  1)  r  (l) 


r  l  pipJu  -^pj' 


*  ■$’  r,r  ■  •«’ 


_L  r(l)  V  (2) 


2  iui  r*  pipj(i-  p4pj>  i<2)  j(1)  p^'i-pipj 

n  i  j  J  AJnij  J  J 


<P  J 


*  J,"’  J,'”  w 

k/j 


Pi) 


2  2  y  (2)  r  (2)  v  (2) 

vnn  L  L- 


2  i  J$i  k/i  PjPkPi(1 
k/j 


PA) 


+  .j.  j.  PjP*Pi(1  •  Pi} 

n  i  j/i  k/i 


k/j 


-  ?(1)  l  U)  J.U)  PjPkPid-Pi' 
1  j/i  k/i 
k/i 


v(2)  y  (2)  y  (2) 

?  1.  PjPkPi 

X  j/i  k/1 

k/j 


Using  now  the  estimates  p^(l 
PiPj(l  -  PiPj)  s  k  /  and  PjPkP±U 
negative  terms  in  (30) : 


P^5  $  k  >  PjP^d  -  P±) 
p^)  <  \  ,  we  may  write. 


(1  -  Pi)  >  - 


4/27  , 
omitting 


13. 


2  n2 
n  Var  K  <  — —  + 


n. 


4n2n.  4n2n, 

X  * 


*  '-in  "i lni-l)  *  <-T7  (^>2 


+  i  <^;>2  "i'"!-11  +  i(K^>2  "2<"2-d 


n>.  n.n,  •*  n  ^  2 

+  +2-j  (— f-)  n.  (n.  -  1)  (n,  -  2) 

4n  2  4n  2  4  nnl  1  1  1 


+  2  *  4  (nn2)  n2(n2  -  1)  (n2  -  2) 


1  2 


+  4  *  "2  r3n1(n1  -  l)n2  +  3n2(n2  -  Dn^ 

n 


(31) 


Denoting  n^/n  =  x  ,  n2/n  =  1  -  x  ,  we  obtain  finally  from 


(31)  : 


Var  K 


-*  2x  (1  -  x)  +  -4  (T  +  if  [x2  +  (1  -  x)  2  3  } 
n  n2  4  27 


+ 


JL  /  (1  -  XL2 

n3  \  4x 


+ 


4(1  -  x) 


which  yields  the  asserted  asymptotic  bound  2n^n2/n3  ,  valid  for 
all  sufficiently  large  n  . 

Maximizing  with  respect  to  x  we  obtain 
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COROLLARY .  As  n  »  ,  we  have  regardless  of  and  n 2 

Var  K  s  an  -  l/2n  .  (32) 

Similarly  as  in  the  preceding  section,  we  have 

PROPOSITION  4  .  Under  condition  (16) ,  the  asymptotic  (as 
n  -*  )  distribution  of  K  is  normal. 

The  proof  is  analogous  to  the  proof  of  Proposition  2  and 
will  be  omitted. 

Propositions  3  and  4  provide  means  of  testing  the  hypothesis 
that  p^  =  p^:  one  may  take  the  random  variable  K  as  the 
test  variable,  and  use  one-sided  critical  region,  large  values 
suggesting  rejection. 


4 .  Applications 

A  typical  application,  for  assessing  the  inter-subject  varia¬ 
bility  of  choice  probabilities,  has  already  been  described  in  the 
Introduction.  Somewhat  more  generally,  one  may  consider  the  follow¬ 
ing  situation. 

Suppose  that  the  subjects  are  trained  to  perform  some  classi¬ 
fication  task,  e.g.,  classify  some  objects  into  binary  categories. 
Taking  one  of  these  categories  as  "success”,  and  classifying  the 
same  set  of  objects  twice  by  the  same  group  of  subjects,  one  obtains 
a  variety  of  possibilities  of  applying  the  results  of  this  paper. 

Let  p.^  be  the  probability  that  i-th  subject  will  classify  j-th 
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object  to  the  chosen  category,  and  let  be  0  or  1 

depending  on  how  the  j-th  object  was  classified  by  the  i-th 
subject  on  the  first  and  second  trial. 

Fixing  i  ,  and  taking  a  large  set  of  objects,  one  can  estimate 
the  variability  of  under  changes  of  j  ;  if  this  variability 

is  large,  one  may  want  to  identify  objects  which  are  easier,  or 
more  difficult,  to  classify. 

On  the  other  hand,  fixing  j  ,  and  varying  i  ,  one  can  test 
whether  there  exists  sufficient  homogeneity  among  classifying  sub¬ 
jects;  one  can  also  test  the  hypothesis,  using  Propositions  3  and 
4,  that  the  two  groups  of  subjects  are  equally  well  trained,  etc. 

If  there  are  more  than  two  categories  in  the  classification, 
one  can  apply  the  same  technique  by  fixing  one  category.  However, 
if  one  can  ensure  not  two,  but  four  independent  classifications 
for  each  person,  then  one  can  define  "success"  as  the  event  "iden¬ 
tical  classifications  on  two  successive  trials". 


Letting  Pijk  »  k  =  1,2,. 


denote  the  probability  that  i-th 


subject  will  classify  j-th  object  into  k-th  category*  the  proba¬ 


bility  of  success  in  this  case  is 


£  pijk  ■  vij  • 

Again,  one  can  test  variability  of  v^j  under  changes  of  sub¬ 
jects  and  objects. 

Finally,  observe  that  the  above  ways  of  testing  the  quality 
of  classification  by  human  classifiers  is  independent  of  the  con¬ 
cept  of  "true  category"  of  the  object,  and  applies  equally  well 


16 


to  classification  schemes  in  which  such  concept  makes  no  sense 
(e-g.  grading  students*  papers,  etc.). 
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